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Subject:  Defense  Acquisitions:  Perspectives  on  Potential  Changes  to  Department  of  Defense 
Acquisition  Management  Framework 

The  National  Defense  Authorization  Act  (NDAA)  for  Fiscal  Year  2008  directed  GAO  to  report 
on  potential  modifications  of  the  organization  and  structure  of  the  Department  of  Defense 
(DOD)  for  Major  Defense  Acquisition  Programs  (MDAP).  In  preparing  the  report,  the  review 
was  required  to  include  the  feasibility  and  advisability  of  seven  potential  modifications  of 
DOD’s  organization  and  structure  for  MDAPs.1  We  were  charged  with  reporting  on  the 
feasibility  and  advisability  of  (1)  establishing  a  process  in  which  the  commanders  of 
combatant  commands  (COCOM)  assess  and  provide  input  on  the  capabilities  needed  to 
successfully  accomplish  their  missions  over  a  long-term  planning  horizon  of  15  years  or 
more;  (2)  establishing  a  materiel  solutions  process  for  addressing  identified  gaps  in  critical 
warfighting  capabilities,  under  which  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  (USD  (AT&L))  circulates  among  the  military  departments  and 
appropriate  defense  agencies  requests  for  proposals  for  technologies  and  systems  to  address 
such  gaps;  (3)  revising  the  acquisition  process  by  establishing  shorter,  more  frequent 
acquisition  program  milestones;  (4)  requiring  the  milestone  decision  authority  (MDA)  to 
specify  at  the  time  of  milestone  B  approval  the  period  of  time  that  will  be  required  to  deliver 
an  initial  operational  capability  (IOC)  to  the  commanders  of  the  relevant  COCOMs;  (5) 
establishing  a  new  office  to  provide  independent  cost  and  performance  estimates;  (6) 
requiring  certifications  of  program  status  to  the  Defense  Acquisition  Executive  (DAE)  and 
Congress  prior  to  milestone  approval;  and  (7)  modifying  the  role  played  by  Chiefs  of  Staff  of 
the  Armed  Forces  in  the  requirements,  resource  allocation,  and  acquisition  processes.2  See 
enclosure  II  for  the  full  wording  of  the  potential  changes. 

In  carrying  out  this  review,  as  Congress  directed,  we  obtained  the  views  of  current  and  prior 
senior  DOD  acquisition  officials,  currently  serving  senior  military  officers  involved  in  setting 
requirements,  and  other  experts  including  some  who  participated  in  previous  reviews  of 
DOD’s  organization  and  structure  for  carrying  out  the  acquisition  of  major  weapon  systems. 3 
We  also  reviewed  and  made  extensive  use  of  GAO’s  prior  work  and  studies  commissioned  by 
DOD  or  the  President — listed  in  enclosure  IV. 


’Pub.  L.  No.  110-181,  §  813  (2008). 

In  assessing  the  seven  potential  changes,  we  defined  feasible  changes  as  those  changes  that  are 
reasonable  to  implement,  and  advisable  changes  as  those  that  could  improve  MDAP  outcomes. 

3DOD  issued  updated  guidance  (DOD  Instruction  5000.02)  concerning  the  operation  of  its  defense 
acquisition  system  in  December  2008.  We  conducted  our  interviews  with  the  experts  prior  to  DOD 
issuing  the  updated  guidance.  We  believe  there  is  nothing  in  the  updated  guidance  that  would  change 
the  opinions  provided  to  us  by  the  experts  or  the  results  of  our  engagement. 
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We  conducted  this  performance  audit  from  January  2008  to  February  2009  in  accordance 
with  generally  accepted  government  auditing  standards.  Those  standards  require  that  we  plan 
and  perform  the  audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis 
for  our  findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit 
objectives.  For  more  on  our  scope  and  methodology,  see  enclosure  I. 

Results  in  Brief 

Based  on  our  discussions  with  acquisition  experts  and  reviews  of  our  own  and  key  studies  on 
DOD  acquisition,  we  believe  that  six  of  the  seven  potential  changes  included  in  the  fiscal  year 
2008  NDAA  could  help  improve  DOD’s  acquisition  processes  with  some  modifications,  but 
only  if  they  are  accompanied  by  additional  changes  in  the  overall  acquisition  environment. 
Specifically,  the  six  changes  are  (1)  establishing  a  process  in  which  the  combatant 
commanders  assess  and  provide  input  on  the  long-term  capabilities  needed  to  successfully 
accomplish  their  missions;  (2)  establishing  a  materiel  solutions  process  for  addressing 
identified  gaps  in  critical  warfighting  capabilities,  under  which  USD  (AT&L)  circulates  among 
the  military  departments  and  appropriate  defense  agencies  requests  for  proposals  for 
technologies  and  systems  to  address  such  gaps;  (3)  revising  the  acquisition  process  by 
establishing  shorter,  more  frequent  acquisition  program  milestones;  (4)  requiring  the  MDA  to 
specify  at  the  time  of  milestone  B  approval  the  period  of  time  that  will  be  required  to  deliver 
an  IOC  to  the  relevant  combatant  commanders;  (5)  establishing  a  new  office  to  provide 
independent  cost  and  performance  estimates;  and  (6)  requiring  certifications  of  program 
status  to  the  DAE  and  Congress  prior  to  milestone  approval.  The  sixth  potential  change 
appears  to  have  been  instituted  already  in  recent  legislation.4  Finally,  we  could  not  determine 
whether  the  remaining  potential  change — modifying  the  role  played  by  Chiefs  of  Staff  of  the 
Armed  Forces  in  the  requirements,  resource  allocation,  and  acquisition  processes — would 
improve  acquisition  outcomes  as  this  potential  change  requires  further  definition  and  more 
study. 

Our  discussions  with  acquisition  experts  indicate  that  these  changes  may  not  achieve  the 
desired  improvement  in  acquisition  outcomes  unless  they  are  accompanied  by  changes  in  the 
overall  acquisition  environment,  its  culture,  and  the  incentives  provided  for  success.  Some  of 
the  DOD  executives  we  interviewed — both  current  and  former — emphatically  stated  that 
these  changes  will  not  improve  the  acquisition  system  until  the  department  can  significantly 
improve  its  prioritization  of  needs  so  that  fewer  programs  are  competing  for  available  dollars. 
Furthermore,  some  of  the  experts  expressed  a  strong  belief  that  none  of  the  potential 
changes  may  be  acceptable  to  some  process  stakeholders  because  some  stakeholders  like  the 
process  as  it  is.  Another  expert  characterized  the  acquisition  process  as  a  system  in 
equilibrium.  It  will  take  strong,  consistent  strategic  leadership  to  change  the  current  culture 
and  environment.  DOD  did  not  provide  a  written  response  to  a  draft  of  this  correspondence, 
but  provided  technical  comments  which  we  have  incorporated  as  appropriate. 

Background 

DOD’s  MDAPs  are  carried  out  under  an  overall  defense  acquisition  management  framework 
comprised  of  three  major  processes:  one  for  requirements  definition,  one  for  resource 
allocation,  and  one  for  the  development  and  production  of  specific  systems.0  Within  that 
framework,  the  requirements  process  identifies,  assesses,  and  prioritizes  warfighting  needs 


4Pub  L.  No.  109-163,  §  801  (2006),  Pub.  L.  No.  109-364,  §  805  (2007),  Pub.  L.  No.  110-181,  §  943  (2008). 
“Federal  statutes,  located  primarily  in  title  10  of  the  United  States  Code,  along  with  the  Federal 
Acquisition  Regulation,  the  Defense  Federal  Acquisition  Regulation  Supplement,  defense  regulation 
and  guidance  provide  the  framework  for  the  defense  acquisition  system. 
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utilizing  DOD’s  Joint  Capabilities  Integration  and  Development  System  (JCIDS);  the  resource 
allocation  process  governs  the  distribution  of  financial  resources  across  DOD  and  the 
military  services  through  DOD’s  Planning,  Programming,  Budgeting,  and  Execution  System 
(PPBES)  to  meet  validated  requirements;  and  the  Defense  Acquisition  System  (DAS)  serves 
to  manage  the  execution  of  product  development  and  procurement. b 

DOD’s  overall  acquisition  management  framework  in  theory  integrates  three  interdependent 
and  stable  processes  of  budget,  acquisition,  and  requirements  and  is  built  on  a  foundation 
that  requires  stability  and  continuity.  Within  that  framework,  the  DAS  is  the  acquisition 
process  that  defines  how  to  buy  weapon  systems,  but  it  does  not  include  requirements  and 
budget.  The  system  involves  contracting,  program  management,  developmental  and 
operational  test  and  evaluation,  production,  and  initial  fielding  of  weapon  systems.  While  all 
these  activities  are  necessary  and  important,  they  are  a  subset  of  the  strategic-level  overall 
management  framework.  Together,  what  DOD  decides  to  buy  at  the  strategic  level  and  how  it 
develops,  tests,  and  produces  weapon  systems  at  the  program  level  determine  the  capabilities 
DOD  delivers  to  the  warfighter. 

DOD  Directive  5000.01  defines  the  DAS  as  the  management  process  used  by  DOD  to  provide 
weapons  systems  and  other  equipment  to  users.  DOD  Instruction  5000.02  provides  more 
specific  guidance  and  identifies  the  entrance  and  exit  criteria  necessary  for  an  acquisition 
program  to  move  through  each  of  the  phases  of  the  acquisition  process.  DOD  recently 
updated  DOD  Instruction  5000.02  in  December  2008  to  incorporate  changes  to  policy  that 
were  the  result  of  new  or  revised  sections  of  public  law  enacted  since  2003;  approved  DOD 
policy  appearing  in  over  25  policy  memos;  and  DOD  responses  to  GAO  and  DOD  Office  of 
Inspector  General  reports.  A  comparison  of  the  new  and  prior  processes  and  an  explanation 
of  the  major  changes  are  provided  in  enclosure  III. 

The  MDA  approves  the  program  structure,  including  the  type  and  number  of  decision  points.' 
Milestone  decision  points  can  initiate  programs  and  authorize  entry  into  the  major  acquisition 
process  phases.  Decision  reviews  assess  progress  and  authorize  or  halt  further  program 
activity.  At  each  milestone,  the  MDA  (which  will  be  the  DAE  or  Component  Acquisition 
Executive  (CAE)  depending  upon  the  scale  or  complexity  of  the  acquisition  program) 
conducts  a  management  review  to  determine  if  a  program  is  ready  to  progress  to  the  next 
phase  of  the  framework.8  Milestone  A  review  is  held  between  the  material  solution  analysis 
and  technology  development  phases;  milestone  B  review  is  held  between  the  technology 
development  and  engineering  and  manufacturing  development  phases;  and  milestone  C 
review  is  held  between  the  engineering  and  manufacturing  development  and  the  production 
and  deployment  phases. 


6JCIDS  falls  under  the  oversight  of  the  Chairman,  Joint  Chiefs  of  Staff  and  is  defined  in  CJCSI 3170. 01F. 
PPBES  falls  under  the  oversight  of  the  Deputy  Secretary  of  Defense  and  is  defined  in  DOD  Financial 
Management  Regulation  7045. 14-R.  DAS  falls  under  the  oversight  of  the  USD  (AT&L)  and  is  defined  in 
DOD  Directive  5000.01  and  DOD  Instruction  5000.02. 

’According  to  DOD,  the  MDA  is  the  designated  individual  who  has  overall  responsibility  for  an 
investment.  This  person  has  the  authority  to  approve  an  investment’s  progression  in  the  acquisition 
process  and  is  responsible  for  reporting  cost,  schedule,  and  performance  results.  For  example,  the 
milestone  decision  authority  for  a  major  defense  acquisition  program,  when  not  delegated  to  the 
component  level,  is  the  USD  (AT&L). 

The  DAE  is  the  USD  (AT&L)  who  has  responsibility  for  supervising  the  DAS.  The  DAE  takes 
precedence  on  all  acquisition  matters  after  the  Secretary  and  the  Deputy  Secretary  of  Defense.  In  the 
military  departments,  the  officials  delegated  as  CAEs  (also  called  Service  Acquisition  Executives 
(SAE))  are  respectively,  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  and  Technology) 
(ASA  (AL&T)),  the  Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition)  (ASN 
(RD&A)),  and  the  Assistant  Secretary  of  the  Air  Force  (Acquisition)  (ASAF  (A)).  The  CAEs  are 
responsible  for  all  acquisition  functions  within  their  components. 
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GAO,  DOD,  and  other  organizations  have  issued  a  number  of  reports  over  the  past  20  or  more 
years  examining  both  the  overall  acquisition  management  framework  and  the  programmatic 
implementation  of  the  DAS  and  making  recommendations  for  improvements.  Five  reports 
seemed  most  relevant  to  the  seven  potential  changes,  the  challenges  facing  defense 
acquisition,  and  ways  to  mitigate  them — the  Quadrennial  Defense  Review,  the  Center  for 
Strategic  and  International  Studies’  (CSIS)  Beyond  Goldwater-Nichols  series  volume  two,  the 
Defense  Science  Board’s  (DSB)  Summer  Study  on  Transformation,  the  Defense  Acquisition 
Performance  Assessment  (DAPA)  and  the  Packard  Commission  report, 9  a  1986  study  which 
became  the  foundation  for  the  Goldwater-Nichols  Department  of  Defense  Reorganization  Act 
of  1986.10 

Potential  Changes  May  Improve  Acquisition  Outcomes 

We  believe  that  six  of  the  seven  potential  changes — some  with  modification — could  improve 
the  acquisition  process  and  its  outcomes  if  accompanied  by  additional  changes  in  the  overall 
acquisition  environment.  The  sixth  potential  change  appears  to  have  been  instituted  already 
in  recent  legislation.11  It  is  unclear  whether  the  potential  change  modifying  the  role  played  by 
Chiefs  of  Staff  of  the  Armed  Forces  in  the  requirements,  resource  allocation,  and  acquisition 
processes  would  improve  acquisition  outcomes  because  it  requires  further  definition  and 
more  study. 

Establishing  a  Process  in  Which  the  Combatant  Commanders  Assess  and  Provide  Input  on 
the  Long-Term  Capabilities  Needed  to  Successfully  Accomplish  Their  Missions 

Based  on  our  work  and  other  major  acquisition  reform  studies,  we  believe  that  enduring 
requirements  reform  will  occur  when  the  combatant  commanders  have  more  influence  over 
the  requirements  process  to  meet  their  priorities.  This  reform  is  necessary  to  create  a  truly 
joint  military  and  lay  the  foundation  for  an  acquisition  process  that  is  responsive  to 
operational  needs. 

COCOMs  are  responsible  for  conducting  combat  operations  and,  ultimately,  ensuring  that  the 
warfighter  has  the  capabilities  needed  to  defeat  threats.  However,  these  commands  currently 
have  a  role  in  the  requirements  process  that  can  be  strengthened.  A  key  vehicle  for  informing 
the  process  is  the  integrated  priority  list  (IPL),  which  is  nonbinding  and  does  not  obligate  the 
military  services,  which  currently  have  primary  responsibility  over  the  requirements  process, 


9DOD,  2006  Quadrennial  Defense  Review  Report  (Washington,  D.C.,  February  2006);  CSIS,  Beyond 
Goldwater-Nichols:  U.S.  Government  and  Defense  Reform  for  a  New  Strategic  Era,  vol.  2 
(Washington,  D.C.,  July  2005);  DSB,  Defense  Science  Board  Summer  Study  on  Transformation:  A 
Progress  Assessment,  vol.  1.  (Washington,  D.C.,  February  2006);  DOD,  Defense  Acquisition 
Performance  Assessment  Project  Report  (Washington,  D.C.,  January  2006);  DOD,  A  Quest  for 
Excellence:  Final  Report  to  the  President  by  the  President’s  Blue  Ribbon  Commission  on  Defense 
Management  (Washington,  D.C.,  June  1986). 

“See  also  the  Goldwater-Nichols  Department  of  Defense  Reorganization  Act  of  1986,  Pub.  L.  No.  99- 
433. 

uPub.  L.  No.  109-163  §  801  (2006);  Pub.  L.  No.  109-364  §  805  (2007),  and  Pub.  L.  No.  110-181  §  943 
(2008). 
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to  fund  any  of  the  COCOM  requirements.12  However,  many  of  the  COCOMs  do  not  believe 
that  their  needs,  which  are  reflected  through  the  IPL  process,  are  sufficiently  addressed 
through  the  department’s  JCIDS  process.  This  has  created  well-known  problems  among  the 
services  including  duplication  in  capabilities  and  persistent  interoperability  problems.  The 
potential  change  would  provide  more  authority  to  COCOMs  to  ensure  their  long-term  needs 
are  met.  We  believe  if  this  potential  change  were  implemented,  a  more  COCOM-focused 
requirements  process  could  improve  joint  warfighting  capabilities  by  ensuring  that  the 
combatant  commanders — the  customers — are  provided  the  appropriate  level  of  input 
regarding  the  capabilities  needed  to  execute  their  missions  rather  than  relying  on  the  military 
services — the  suppliers — to  drive  requirements.  Our  review  of  DOD’s  past  studies  and  our 
own  work  provides  additional  rationale  for  this  change.  Previous  recommendations  and 
evidence  to  support  this  change  include  the  following: 

•  GAO  recently  reported  the  results  of  our  review  of  documentation  related  to 
proposals  for  new  capabilities  that  found  that  most — almost  70  percent — were 
sponsored  by  the  military  services,  with  little  involvement  from  COCOMs.13 

•  The  DAPA  panel  recommended  that  the  COCOMs  play  the  lead  role  in  defining 
needed  capabilities  and  identifying  gaps,  and  that  the  military  services  and  DOD 
agencies  compete  to  provide  solutions  once  those  gaps  are  identified.  The  report 
further  stated  that  decisions  concerning  which  capabilities  to  acquire  should  be  made 
from  a  joint  operational  perspective.  It  also  found  that  within  the  current 
requirements  identification  process,  the  military  services  provide  the  primary  input 
concerning  needed  capabilities  and  control  the  resources  for  procuring  capabilities, 
thus  the  requirements  frequently  are  not  linked  to  the  capabilities  desired  by  the 
combatant  commanders. 

•  CSIS’s  Beyond  Goldwater-Nichols  body  of  work  and  DSB’s  2005  Summer  Study  on 
Transformation  found  that  only  COCOMs  have  operational  requirements  and 
operational  responsibility  to  employ  the  military  services  as  a  joint  team  and 
therefore  should  have  more  influence  in  establishing  requirements. 

A  small  majority  of  the  experts  we  interviewed  who  commented  on  this  change  (23  of  43) 
believed  that  a  requirements  process  that  provides  the  COCOMs  with  more  influence  over 
which  needs  and  gaps  get  filled  could  improve  acquisition  outcomes  if  the  COCOMs  are 
provided  needed  personnel  and  financial  resources  to  do  this.  Additionally,  some  experts 
stated  that  this  change  would  promote  joint  solutions  and  would  be  more  responsive  to 
warfighters’  most  urgent  needs.  The  remaining  experts,  however,  told  us  the  potential  change 
would  not  improve  the  process,  and  some  believed  this  was  because  the  existing  JCIDS 
process  already  considers  the  combatant  commanders’  long-term  needs  and  that  they  lack 
the  required  expertise  and  resources. 


12The  IPL  defines  the  COCOM’s  highest  priority  capability  gaps  for  the  near  term,  including  shortfalls 
that  may  adversely  affect  COCOM  missions.  These  lists  are  prepared  by  the  COCOMs  and  submitted  to 
the  Joint  Chiefs  of  Staff  and  then  to  the  Joint  Requirements  Oversight  Council  (JROC).  After  JROC 
reviews  the  IPL  and  decides  to  endorse  the  requirements,  the  approved  document  is  given  to  the  Vice 
Chairman  of  the  Joint  Chiefs  and  finally  the  Chairman  of  the  Joint  Chiefs  for  final  review.  The 
document  is  analyzed  against  established  programs  within  the  services  to  see  if  needs  can  be  met 
through  existing  programs.  If  a  capability  is  needed  that  does  not  have  a  corresponding  program  in 
development,  then  the  Joint  Staff  will  determine  the  most  appropriate  organization  to  further  develop 
the  requirement. 

13GAO,  Defense  Acquisitions:  DOD’s  Requirements  Determination  Process  Has  Not  Been  Effective  in 
Prioritizing  Joint  Capabilities,  GAO-08-1060.  (Washington,  D.C.:  Sept.  25,  2008). 
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To  implement  a  more  COCOM-focused  requirements  process,  DOD  would  need  to  ensure  the 
COCOMs  have  sufficient  additional  resources  to  establish  robust  analytical  capabilities  to 
identify  and  assess  their  long-term  requirements.  Also,  while  some  of  the  participants  in 
DOD’s  acquisition  system  may  object  to  a  perceived  shift  in  focus  for  the  COCOMs  from 
short-term  to  long-term  views,  this  objection  may  be  addressed  by  implementing  a  Beyond 
Goldwater-Nichols  Phase  II  Report  recommendation  that  DOD  utilize  functional  commands, 
such  as  the  Joint  Forces  Command,  to  provide  robust  analytical  support  to  the  geographical 
COCOMs  to  identify  and  assess  each  command’s  long-term  requirements.  According  to  DOD, 
it  has  taken  action  already  that  it  believes  is  in  line  with  the  Beyond  Goldwater-Nichols 
Phase  II  Report  recommendation.  A  recent  JROC  memorandum  directed  that  the  lead 
functional  COCOM  for  the  battlespace  awareness,  command  and  control,  logistics,  and  net- 
centric  portfolios  will  coordinate  on  any  capability  documents,  and  the  lead  functional 
combatant  command  Deputy  Commander  will  coordinate  on  the  associated  JROC  or  Joint 
Capability  Board  memoranda  that  validate  those  capability  documents.14  We  believe  this  is  a 
step  in  the  right  direction. 

In  its  latest  Defense  Acquisition  Transformation  Report  to  Congress,  DOD  indicated  that  the 
Joint  Staff  is  heavily  engaging  the  COCOMs  in  the  requirements  process  through  participation 
in  JROC  meetings.  DOD  noted  that  in  fiscal  year  2008  over  75  percent  of  JROC  meetings 
included  one  or  more  COCOM  general  officer  representatives.  Furthermore,  JROC  used  the 
COCOMs’  IPLs  as  the  starting  point  for  a  series  of  assessments  that  identified  the  most 
critical  military  issues  and  prioritized  the  capability  gaps  for  each  Joint  Capability  Area 
(JCA).15  Additionally,  twice  a  year  JROC  visits  and  consults  with  each  COCOM  to  better 
understand  their  warfighting  needs  and  to  provide  feedback  on  JROC’s  and  the  military 
services’  efforts  to  satisfy  those  needs. 

Establishing  a  Materiel  Solutions  Process  for  Addressing  Identified  Gaps  in  Critical 
Warfighting  Capabilities.  Under  Which  USD  fAT&Lj  Circulates  Among  the  Military 
Departments  and  Appropriate  Defense  Agencies  Requests  for  Proposals  for  Technologies  and 
Systems  to  Address  Such  Gaps 

If  other  important  changes  also  take  place,  we  believe  this  potential  change  could 
significantly  improve  acquisition  outcomes.  If  implemented  in  conjunction  with  an  integrated, 
portfolio  management  based  investment  strategy,  it  would  establish  a  single  point  of 
accountability  that  has  the  potential  to  help  ensure  interoperability  among  multiple  systems; 
help  reduce  stovepiped  and  sometimes  duplicative  acquisition  programs;  and  enable  DOD  to 
more  easily  align  its  portfolio  of  systems  within  its  budgetary  resources. 

GAO  and  others  have  time  and  again  reported  on  service  parochialism  in  DOD’s  requirements 
process  and  the  military  services  may  not  be  in  the  best  position  to  decide  what  the 
combatant  commanders  need  to  conduct  their  operations,  ensure  systems  are  interoperable, 
and  eliminate  duplication  of  systems.  Previous  recommendations  and  evidence  to  support 
this  change  include  the  following: 

•  GAO  recently  reported  that  the  JCIDS  process  has  not  yet  met  its  objective  to  identify 
and  prioritize  warfighting  needs  from  a  joint  capabilities  perspective.  Instead, 
capabilities  continue  to  be  driven  primarily  by  the  individual  military  services.  GAO 


14The  Joint  Staff,  Joint  Requirements  Oversight  Council.  JROCM  130-08,  June  20,  2008.  Assignment  of 
Joint  Potential  Designators  and  Coordination  by  Combatant  Commands  o?i  Capabilities 
Documents. 

1BJCA  provide  a  common  medium  to  discuss  and  describe  capabilities  across  many  DOD  activities  and 
processes.  For  example,  Battle  Space  Awareness  is  the  Joint  Capability  Area  that  encompasses  the 
ability  to  understand  dispositions  and  intentions  as  well  as  characteristics  and  conditions  of  the 
operational  environment  that  bear  on  national  and  military  decision  making. 
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has  repeatedly  reported  that  DOD  could  improve  acquisition  outcomes  if,  at  the 
Office  of  the  Secretary  of  Defense  (OSD)  level,  it  utilized  a  portfolio  management 
approach  to  guide  its  investment  strategy  rather  than  rely  on  each  military  service  to 
choose  which  weapons  to  procure. 

•  The  DAPA  panel  recommended  requiring  the  military  services  to  compete  to  satisfy 
the  combatant  commanders’  requirements. 

•  DSB’s  2005  Summer  Study  on  Transformation  recommended  an  organizational 
structure  that  clearly  identifies  the  Secretary  of  Defense  with  support  from  USD 
(AT&L)  as  the  decision  maker  for  weapon  system  solutions  with  advice  from  the 
Chairman  of  the  Joint  Chiefs  of  Staff  and  the  Joint  Staff. 

Of  the  49  experts  we  interviewed  who  commented  on  this  potential  change,  35  believed  that 
it  would  not  improve  the  acquisition  processes.  Some  of  those  experts  told  us  that  USD 
(AT&L)  was  not  properly  resourced  for  this  change  to  be  effective.  Also,  there  appeared  to  be 
uncertainty  among  the  experts  as  to  how  this  change  would  be  implemented.  Some  agreed 
that  the  services  would  be  strictly  tasked  with  providing  solutions  to  requirements  that  have 
already  been  vetted,  while  others  seemed  to  interpret  the  change  as  one  that  would  include 
requirement-setting  responsibility.  However,  the  remaining  experts  felt  this  potential  change 
would  improve  the  process.  Some  of  those  experts  cited  benefits  including  reducing  service 
parochialism  and  increasing  the  role  of  OSD  staff  and  the  combatant  commanders  in  the 
requirements  process. 

To  be  effective,  this  potential  change  should  be  considered  in  conjunction  with  the  potential 
change  to  establish  a  process  in  which  the  combatant  commanders  are  given  more 
responsibility  for  assessing  capabilities  and  determining  requirements.  Together,  these 
changes  could  help  ensure  focus  on  filling  the  capability  gaps  identified  by  the  combatant 
commanders  and  reducing  service  parochialism. 

The  department  has  reported  efforts  that  may  help  in  this  regard.  In  its  latest  Defense 
Acquisition  Transformation  Report  to  Congress,  DOD  stated  that  it  is  addressing  some  of 
the  perceived  problems  in  this  area  by  adopting  concept  decision  and  capability  portfolio 
management  processes  under  the  auspices  of  the  USD  (ATL) — that  are  intended  to  establish 
better  OSD  leadership  to  improve  strategic  investment  decisions.16  DOD  also  stated  that  it  is 
seeking  to  challenge  and  empower  its  acquisition  workforce  to  yield  the  best  value  for  the 
taxpayer  and  capability  for  the  warfighter. 

Revising  the  Acquisition  Process  by  Establishing  Shorter.  More  Frequent  Acquisition 
Program  Milestones 

This  potential  change,  if  adopted  in  a  manner  consistent  with  the  mandate  and  its  supporting 
conference  report,  would  require  shorter,  more  frequent  acquisition  milestones  to  manage 
requirements,  technology,  design,  and  manufacturing  risk  more  closely.1'  Based  on  our  own 
body  of  work  and  review  of  relevant  DOD  studies,  we  believe  a  requirement  for  four 
milestones  would  serve  the  purposes  of  this  potential  change.  In  our  opinion,  this  potential 
change  could  improve  program  execution  if  the  milestones  are  specified  and  aligned  with 
demonstrating  knowledge  at  the  appropriate  time  and  if  they  are  tied  to  certifications  to  the 
DAE. 


“Concept  decision  (now  the  Material  Development  Decision  under  the  updated  5000.02  instruction)  is 
the  first  decision  point  of  the  DAS.  It  authorizes  entry  into  the  material  solution  analysis  phase. 

The  Conference  Report  for  the  National  Defense  Authorization  Act  for  Fiscal  Year  2008  proposes 
seven  milestones.  H.R.  Conf.  Rep.  No.  110-477,  at  951  (2007). 
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GAO  has  previously  reported  that  the  acquisition  management  framework  allows  programs 
to  move  forward  without  adequate  knowledge  at  critical  junctures.  DOD  programs  regularly 
skipped  a  pre-acquisition  milestone  A,  passed  milestone  B  with  limited  systems  engineering 
knowledge  and  critical  technologies  that  were  immature  and  not  ready  for  product 
development.  Programs  do  not  undergo  another  milestone  review  until  production  begins  at 
milestone  C,  sometimes  up  to  10  years  later.  These  practices  left  programs  with  limited 
knowledge  and  lengthy  time  gaps  between  critical  reviews  and  few  opportunities  for  crucial 
management  oversight  and  decision  making  concerning  the  department’s  investment  and  the 
warfighter’s  continuing  need.  DOD  issued  updated  guidance  concerning  the  operation  of  its 
DAS  in  December  2008  to  help  address  these  issues.  However,  it  is  too  soon  to  determine 
how  successful  these  changes  will  be  in  addressing  those  issues. 

GAO  has  advocated  that  OSD  exert  stronger  leadership  in  managing  DOD’s  acquisition 
programs  and  has  specifically  recommended  that  OSD  (1)  develop  and  implement  an 
integrated  enterprise-level  investment  strategy  that  is  based  on  a  joint  assessment  of 
warfighting  needs  and  a  full  set  of  potential  and  viable  alternative  solutions  within  realistic 
and  affordable  budget  projections;  (2)  ensure  DOD’s  acquisition  programs  are  established 
with  sound  business  cases  prior  to  entering  systems  development;  (3)  establish  a  knowledge- 
based  acquisition  process;  and  (4)  improve  leadership  and  accountability. 

DOD’s  recently  revised  acquisition  policy  addresses  some  of  these  needed  changes  to  some 
degree;  for  example,  the  policy  now  encourages  the  development  of  knowledge  earlier  in  the 
acquisition  process.  However,  consistent  with  our  prior  work,  we  believe  an  additional 
milestone  review  is  needed  midway  through  engineering  and  manufacturing  development — 
during  the  product  development  phase — to  show  the  design  is  stable,  and  that  adequate 
knowledge  exists  and  demonstrates  the  weapon  system  design  will  perform  as  expected.  We 
believe  this  is  necessary  to  reduce  risk  and  optimize  the  department’s  investment.  According 
to  DOD,  the  recently  revised  process  achieves  this.  In  addition  to  the  three  milestone 
reviews,  the  revised  instruction  requires  a  formal  MDA  decision  review — the  Post  Critical 
Design  Review  Assessment — during  the  Engineering  and  Manufacturing  Development  Phase. 
Additionally,  if  a  program  did  not  conduct  a  Preliminary  Design  Review  prior  to  the  milestone 
B  decision,  the  MDA  will  conduct  a  Post  Preliminary  Design  Review  Assessment  following 
the  Preliminary  Design  Review.  According  to  DOD,  these  MDA  assessments  are  designed  to 
achieve  the  same  substantive  benefit  as  a  milestone  review,  and  will  result  in  decisions  that 
will  be  documented  in  acquisition  decision  memoranda.  Figure  1  compares  DOD’s  revised 
acquisition  process  and  GAO’s  recommended  knowledge-based,  evolutionary  process. 
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Figure  1:  Comparison  of  DOD’s  Acquisition  Process  and  GAO’s  Knowledge-Based  Acquisition  Process 
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Source:  GAO  analysis  of  DOD's  acquisition  process  and  GAO's  best  practices. 

GAO’s  work  has  repeatedly  demonstrated  the  need  for  DOD  to  follow  a  knowledge-based 
acquisition  process  and  to  develop  knowledge  earlier  in  the  acquisition  process,  basing  the 
decision  to  enter  the  system  development  phase  of  the  acquisition  process  with  preliminary 
design  knowledge  in  hand.  We  agree  with  the  department’s  new  policy  that  prescribes 
conducting  milestones  A,  B,  and  C.  We  believe  a  fourth  milestone,  centered  at  each 
program’s  Post  Critical  Design  Review  Assessment  midway  through  development,  is  also 
needed.  With  that  in  mind,  each  program  would  have  to  pass  through  the  following  milestone 
reviews: 

•  Milestone  A — A  review  to  determine  the  weapon  system  concept  and  identify  the 
technologies  available  to  meet  a  validated  requirement  for  a  new  weapon  system; 


Milestone  B  (equivalent  to  Knowledge  Point  1) — A  review  to  determine  through 
prototyping  and  disciplined  and  robust  systems  engineering  practices  that  a  match  is 
made  between  requirements  and  available  mature  technology;  funding,  and  other 
resources; 
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•  Milestone  C  (equivalent  to  Knowledge  Point  3) — A  review  to  demonstrate  that 
manufacturing  processes  are  in  control  and  that  the  system  can  be  produced  within 
cost,  schedule,  and  quantity  targets. 

Consequently,  shorter,  more  frequent  acquisition  program  milestones  could  improve  DOD’s 
accountability  and  the  acquisition  process  by  providing  OSD’s  decision  makers  with 
improved  data  upon  which  to  make  decisions  and  more  frequent  opportunities  to  exercise 
oversight  over  these  expensive  investments.  Recommendations  from  prior  studies  on  DOD 
acquisition  support  realignment  along  these  lines  and  include: 

•  The  DAPA  panel  reported  that  the  chief  risk  reduction  opportunities  exist  between 
milestone  A  and  milestone  B.  Nonetheless,  DOD  focuses  much  of  its  efforts  on 
reaching  and  passing  milestone  B,  which  leads  to  starting  programs  with  immature 
technologies  and  a  long  cycle  of  program  instability,  requirements  and  budget 
changes,  schedule  delays,  and  repeated  rebaselining.18  The  DAPA  panel  recommended 
DOD  realign  the  milestone  B  decision  to  occur  at  preliminary  design  review  to  reduce 
program  risk  by  increasing  knowledge  prior  to  milestone  B. 

•  The  Institute  for  Defense  Analysis  (IDA)  in  its  Acquisition  Initiatives — Phase  II  report 
recommended  USD  (AT&L)  should  reinstitute  MDA  oversight  of  the  milestone  A 
process  so  that  a  sound  systems  engineering  management  plan  is  put  in  place  for  each 
alternative  that  will  be  pursued  into  a  formal  risk  reduction/demonstratiomValidation 
phase,  leading  to  a  possible  milestone  B  formal  initiation  of  a  major  new  acquisition 
program.19  The  report  also  suggested  there  should  be  a  very  high  bar  for  waivers  of  an 
inadequate  systems  engineering  plan  at  formal  program  initiation  at  milestone  B. 

Regarding  this  potential  change,  28  of  the  52  experts  we  interviewed  who  commented  on  this 
potential  change  believed  that  it  would  not  improve  the  process.  Some  of  the  experts  who 
told  us  the  potential  change  would  not  improve  the  acquisition  process  indicated  that  the 
current  process  and  oversight  reviews  are  adequate  and  noted  that  more  reviews  do  not 
always  equate  to  improved  outcomes.  However,  many  of  the  experts  we  interviewed  also  told 
us  that  this  potential  change  could  improve  the  process  depending  upon  how  it  is 
implemented.  Some  of  the  experts  told  us  that  additional  milestones  should  be  accompanied 
with  a  more  uniform  data  gathering  process,  quantifiable  metrics,  significantly  more  systems 
engineering  knowledge,  and  incremental  development  strategies. 

We  believe  DOD’s  updated  DOD  Instruction  5000.02  aligns  with  the  intent  of  this  potential 
change  and  GAO’s  best  practices  model.  However,  to  be  more  effective,  we  believe  the 
revised  policy  should  be  implemented  in  conjunction  with  the  potential  change  to  require  the 
milestone  decision  authority  to  specify,  at  the  time  of  milestone  B  approval,  the  period  that 
will  be  required  to  deliver  IOC  to  the  relevant  combatant  commanders,  which  is  discussed  in 
the  following  report  segment.  In  combination,  these  potential  changes  should  result  in  more 
frequent  milestone  reviews  with  shorter  timeframes  between  milestones — both  as  a  result  of 
the  adoption  of  the  milestone  B’  and  the  adoption  of  a  more  incremental,  evolutionary 
acquisition  approach — helping  to  focus  development  on  more  manageable,  less  aggressive 
increments  of  capability  and  facilitating  accountability.  Additionally,  strong  consistent 
leadership  that  demands  and  rewards  the  proper  use  of  knowledge  by  programs  is  needed. 


“According  to  the  GAO  Cost  Assessment  Guide,  rebaselining  should  occur  rarely,  as  infrequently  as 
once  in  the  life  of  a  program  or  project.  Schedule  rebaselining  should  occur  only  when  a  schedule 
variance  is  significant  enough  to  limit  its  utility  as  a  predictor  of  future  schedule  performance.  GAO, 
Cost  Assessment  Guide:  Best  Practices  for  Estimating  and  Managing  Program  Costs,  Exposure 
Draft,  GAO-07-1 134SP  (Washington,  D.C.:  July  2007),  p.  251. 

19IDA,  Acquisition  Initiatives  Review,  Phase  II,  (Alexandria,  VA.:  February  2006). 
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In  its  latest  Defense  Acquisition  Transformation  Report  to  Congress,  DOD  states  its 
acquisition  policy  encourages  the  development  of  knowledge  earlier  in  the  acquisition 
process  and  has  placed  clear  emphasis  on  the  need  for  more  systems  engineering  analysis 
and  prototyping  earlier  in  the  process.  For  example,  the  policy  now  requires  that  programs 
involving  immature  technologies  undergo  a  milestone  A  review  and  requires  a  disciplined 
preliminary  design  review  prior  to  or  soon  after  the  start  of  a  program  at  milestone  B.  While 
the  recently  updated  policy  does  not  create  new  milestones  for  design  demonstration,  it 
aligns  existing  milestones  and  other  decision  points  with  obtaining  critical  knowledge. 

Requiring  the  MPA  to  Specify  at  the  Time  of  Milestone  B  Approval  the  Period  of  Time  That 
Will  Be  Required  to  Deliver  an  IOC  to  the  Relevant  Combatant  Commanders 

We  believe  that  this  potential  change,  if  modified  to  require  a  time-certain  development 
period  not  to  exceed  6  years  would  significantly  improve  acquisition  outcomes.20  However,  it 
can  only  be  successful  if  the  department  uses  an  incremental,  evolutionary,  and  knowledge- 
based  approach  to  weapon  system  acquisition  programs.  New  programs  would  attempt  to 
develop  and  produce  capabilities  that  are  achievable  within  6  years,  requiring  a  new  program 
to  achieve  future  incremental  capabilities.  Limiting  development  to  5-  or  6-year  increments 
(or  shorter  increments)  would  force  smaller,  more  manageable  commitments  in  capabilities 
for  each  increment  and  make  costs  and  schedules  more  predictable.  In  addition,  it  could  help 
to  stabilize  funding  by  reducing  the  need  to  make  up  for  program  deviations. 

In  today’s  acquisition  environment,  it  sometimes  takes  more  than  a  decade  to  produce  an 
initial  capability.  As  a  result,  programs  face  increased  risks  that  circumstances  will  change 
around  them — for  example,  technology  changes,  requirements  changes,  and  budget 
pressures — driving  up  costs  and  delaying  fielding.21  Requiring  a  milestone  decision  authority 
to  specify  the  period  required  to  deliver  an  IOC,  essentially  incorporating  a  time-certain 
philosophy,  could  help  DOD  deliver  useful  military  capabilities  in  a  more  timely  manner. 
Previous  recommendations  and  evidence  to  support  this  change  include  the  following: 

•  GAO  recommended  in  2007  that  DOD  take  decisive  actions  to  shorten  cycle  times  in 
delivering  needed  combat  capabilities  to  the  warfighter  to  include  adopting  a  time- 
certain  development  cycle  that  can  deliver  an  increment  of  new  capability  within  5  to 
6  years  after  the  start  of  system  design  and  development22  We  have  cited  past 
acquisitions,  such  as  the  F-16  and  Small  Diameter  Bomb,  as  examples  of  successful 
incremental  development. 

•  The  DAPA  panel  recommended  that  DOD  make  program  schedules  a  key 
performance  parameter  as  well  as  shift  its  acquisition  approach  to  incorporate  a  time- 
certain  development  strategy.  Inserting  time  as  a  key  performance  parameter  would 
encourage  evolutionary  development  and  force  more  detailed  up-front  requirements 
statements. 


20 According  to  DOD,  a  time-certain  development  period  should  be  tailored  to  the  characteristics  of  the 
weapon  system  under  development.  For  example,  6  years  to  develop  and  produce  major  acquisitions, 
such  as  a  nuclear  submarine  or  an  aircraft  carrier,  may  not  be  realistic. 

21CJCSI 3170. OIF,  May  1,  2007,  requires  the  Capability  Development  Document  (CDD)  and  Capability 
Production  Document  (CPD)  to  include  the  definition  of  IOC  and  the  required  IOC  date.  The  CDD  and 
CPD  are  approved  and  validated  by  the  JROC,  and  the  Acquisition  Program  Baseline  (APB)  is 
approved  by  the  MDA.  DODI  5000.02,  December  8,  2008,  requires  the  APB  to  include  the  definition  of 
system  IOC  and  the  IOC  date.  However,  no  time  limitations  are  specified  in  either  instruction. 

"GAO,  Tactical  Aircraft:  DOD  Needs  a  Joint  and  Integrated  Investment  Strategy,  GAO-07-415 
(Washington,  D.C.:  Apr.  2,  2007). 
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•  The  Beyond  Goldwater-Nichols  work  found  that  current  initiatives  may  prove  fruitful 
in  the  next  few  years,  for  example  more  aggressively  managing  risk  at  the  early  stages 
of  the  development  process  by  assessing  technology  maturity  more  accurately  and 
giving  increased  weight  to  getting  capabilities  into  the  field  faster. 

•  The  Packard  Commission  reported  that  an  unreasonably  long  acquisition  cycle — 10  to 
15  years  for  major  weapon  systems — is  the  central  problem  from  which  most  other 
acquisition  problems  stem.  It  leads  to  unnecessarily  high  costs  of  development — a  10 
year  acquisition  cycle  is  clearly  more  expensive  than  a  5  year  cycle.  It  also  leads  to 
obsolete  technology  in  fielded  equipment.  Because  long-term  forecasts  are  uncertain 
at  best,  users  tend  to  err  on  the  side  of  overstating  the  threat. 

More  than  half  of  the  experts  we  interviewed  who  commented  on  this  potential  change  (26  of 
49)  believed  it  could  improve  the  acquisition  process.  Several  of  those  experts  told  us  this 
change  could  be  effective  if  it  is  aligned  with  evolutionary  acquisition  and  time-certain 
development  principles.  Several  of  the  experts  who  commented  that  this  potential  change 
may  not  improve  acquisition  results  told  us  that  historically,  DOD  program  managers  (PM)  do 
not  control  requirements  and  resources  or  cost  estimates  are  not  realistic. 

To  be  most  effective,  we  believe  this  potential  change  should  be  implemented  in  conjunction 
with  the  potential  change  to  revise  the  acquisition  process  by  establishing  shorter,  more 
frequent  acquisition  program  milestones.  Furthermore,  the  success  of  implementing  time- 
certain  development  may  be  contingent  upon  several  factors.  For  example,  PMs  must  be 
empowered  to  control  their  program  and  defer  additional  requirements  until  a  future 
increment;  there  must  be  unity  among  leadership  in  support  of  the  program;  and  PM  tenure 
agreements  need  to  bridge  the  gap  between  major  program  milestones  to  facilitate 
accountability. 

In  its  latest  Defense  Acquisition  Transformation  Report  to  Congress,  DOD  agreed  with  the 
intent,  but  not  the  language,  of  the  DAPA  panel’s  findings  on  time-certain  acquisition  and 
stated  that  it  was  implementing  that  intent  by  selecting  time-defined  approaches  consistent 
with  the  users’  needs  at  the  concept  decision  point  in  the  acquisition  process. 

Establishing  a  New  Office  to  Provide  Independent  Cost  and  Performance  Estimates 

We  believe  that  establishing  an  independent  assessment  office  could  enhance  the  cost 
estimates  produced  if  the  organization  is  required  to  report  the  results  of  knowledge-based 
cost  assessments  for  MDAPs  directly  to  Congress  and  the  Secretary  of  Defense,  and  is  given 
the  resources  needed  to  develop  thorough  estimates. 

The  Cost  Analysis  Improvement  Group  (CAIG)  is  the  organization  currently  responsible  for 
conducting  independent  cost  assessments  for  MDAPs  and  reports  to  the  department’s 
program  analysis  and  evaluation  organization  (PA&E).2’  However,  USD  (AT&L) — through 
oversight  of  the  DAS — is  the  ultimate  customer  for  CAIG  estimates,  and  ultimately  decides 
whether  the  CAIG  independent  cost  estimate  will  be  used  as  the  basis  to  fund  the  program.24 
Establishing  an  office  of  independent  assessment  that  reports  its  results  directly  to  Secretary 
of  Defense  and  Congress,  similar  to  the  office  of  the  Director  of  Operational  Test  and 
Evaluation  (DOT&E)  would  more  closely  align  the  CAIG  with  the  Secretary  of  Defense  and 


23DOD  Directive  5000.04-Cost  Analysis  Improvement  Group,  dated  August  16,  2006,  describes  the 
responsibilities  of  the  CAIG. 

24 According  to  DOD,  the  CAIG  prepares  a  signature  report  on  each  of  its  estimates,  and  provides  this 
report  directly  to  the  USD  (AT&L).  The  CAIG  independent  cost  estimates  are  prepared  in  accordance 
with  the  existing  statutory  requirements  10  USC  §  2434,  and  it  is  presented  for  consideration  at  the 
Defense  Acquisition  Board  milestone  review  meetings. 
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USD  (AT&L)  providing  a  new  level  of  accountability  and  the  opportunity  to  be  more  fully 
integrated  into  the  acquisition  management  framework.25  DOD’s  inability  to  allocate  funding 
effectively  to  programs  is  largely  driven  by  its  acceptance  of  unrealistic  cost  estimates  and  a 
failure  to  balance  needs  based  on  available  resources.  The  uniqueness  of  each  program,  the 
lack  of  sufficient  knowledge  about  system  requirements,  technology  and  design  maturity,  and 
the  limited  analytical  tools  available  are  often  cited  as  factors  that  contribute  to  optimistic 
forecasts  of  development  costs.  Previous  recommendations  and  evidence  to  support  this 
change  include  the  following: 

•  GAO  has  reported: 

•  Development  costs  for  major  acquisition  programs  are  often  underestimated 
at  program  initiation  sometimes  by  30  to  40  percent.25  Additionally,  weapon 
system  programs  are  initiated  without  sufficient  knowledge  about  system 
requirements,  technology,  and  design  maturity  and  that  lacking  such 
knowledge,  managers  rely  on  assumptions  that  are  consistently  too  optimistic, 
exposing  programs  to  unnecessary  risks  and  ultimately  cost  growth  and 
schedule  delays.2' 

•  The  policy  concerning  independent  cost  estimates  does  not  require  that  the 
estimates  be  relied  upon  to  set  budgets,  only  that  they  be  considered  at  key 
acquisition  decision  points.2S  Additionally  in  2008,  we  reported  that  for  many 
of  the  20  major  weapons  programs  we  reviewed,  the  initial  approved 
development  baseline  funding  was  lower  than  the  CAIG’s  cost  estimate.29 

•  The  independence  of  DOT&E  and  its  resulting  authority  to  report  directly  to 
Congress  is  the  foundation  of  its  effectiveness.  That  independence  along  with 
its  legislative  mandate  provides  sufficient  freedom  and  authority  to  exercise 
effective  oversight  of  the  operational  testing  and  evaluation  of  new  systems.30 

•  The  DAPA  panel  recommended  adjusting  program  estimates  to  reflect  high 
confidence  levels — defined  as  a  program  with  an  80  percent  chance  of  completing 
development  at  or  below  estimated  cost. 

Overall,  46  of  the  49  experts  we  interviewed  believe  a  new  office  to  provide  cost  and 
performance  estimates  is  not  needed.  Many  of  the  experts  we  interviewed  believe  the  CAIG  is 
independent.  Some  suggested  that  the  lack  of  resources  within  the  CAIG,  inadequate  systems 
engineering,  long  acquisition  time  frames,  or  adoption  of  immature  technologies  contribute 
to  poor  cost  estimates.  Additionally,  according  to  one  CAIG  official,  its  cost  estimates  and 
independence  could  be  improved  if  the  organization  had  more  personnel  resources  with  the 


25DOT&E  is  the  principal  staff  assistant  and  senior  advisor  to  the  Secretary  of  Defense  on  operational 
test  and  evaluation  (OT&E)  in  DOD.  DOT&E’s  responsibilities  include  issuing  DOD  OT&E  policy  and 
procedures;  reviewing  and  analyzing  the  results  of  OT&E  conducted  for  each  MDAP;  providing 
independent  assessments  to  Secretary  of  Defense,  the  USD  (AT&L),  and  Congress;  making  budgetary 
and  financial  recommendations  to  the  Secretary  of  Defense  regarding  OT&E;  and  providing  oversight 
to  ensure  OT&E  for  MDAPs  is  adequate  to  confirm  operational  effectiveness  and  suitability  of  the 
defense  system  in  combat  use. 

2bGAO,  Defense  Acquisitions:  A  Knowledge-Based,  Funding  Approach  Could  Improve  Major  Weapon 
System  Program  Outcomes,  GAO-08-619  (Washington,  D.C.:  July  2,  2008). 

2‘GAO,  Defense  Acquisitions:  Better  Weapon  Program  Outcomes  Require  Discipline,  Accountability, 
and  Fundamental  Changes  in  the  Acquisition  Environment,  GAO-08-782T  (Washington,  D.C.:  June  3, 
2008). 

2SGAO,  Space  Acquisitions:  DOD  Needs  to  Take  More  Action  to  Address  Unrealistic  Initial  Cost 
Estimates  of  Space  Systems,  GAO-07-96  (Washington,  D.C.:  November  17,  2006). 

29GAO-08-619. 

' 10 GAO,  Test  and  Evaluation:  Impact  of  DOD’s  Office  of  the  Director  of  Operational  Test  and 
Evaluation,  NSIAD-98-22  (Washington,  D.C.:  October  24,  1997). 
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appropriate  systems  engineering  expertise  to  conduct  independent  systems  engineering 
performance  reviews  that  could  then  be  used  to  prepare  improved  cost  estimates  and  risk 
determinations. 

It  is  critical  to  note  that  for  this  change  to  be  effective  and  result  in  more  realistic,  reliable, 
and  independent  cost  estimates,  we  believe  DOD  needs  to  improve  its  cost  estimating 
through  a  disciplined,  knowledge-based  approach  that  reduces  risk  and  uncertainty  over  time 
by  conducting  detailed  systems  engineering  early  in  weapons  systems  development 
processes,  shortening  acquisition  periods,  and  adopting  more  mature  technologies.  Although 
many  experts  believed  that  this  change  would  not  improve  the  acquisition  process,  we 
believe  that  establishing  an  independent  assessment  office  that  provides  independent  cost 
estimates  and  reports  its  results  directly  to  the  Secretary  of  Defense  and  to  Congress  would 
enhance  independence  by  better  informing  stakeholders  outside  of  the  DOD  acquisition 
community.  We  see  no  reason  why  the  CAIG  couldn’t  form  the  basis  of  the  suggested 
organization.  Finally,  that  office  should  have  the  appropriate  resources  to  conduct  such 
knowledge-based  cost  assessments. 

In  its  latest  Defense  Acquisition  Transformation  Report  to  Congress,  DOD  stated  that  USD 
(AT&L)  is  focusing  on  prototyping  to  reduce  technical  risk  and  inform  costs  estimates.  DOD 
is  also  working  to  improve  control  over  developmental  timelines  through  reduced  risk,  better 
knowledge,  and  technical  maturity. 

Requiring  Certifications  of  Program  Status  to  the  DAE  and  Congress  Prior  to  Milestone 

Approvals 

The  National  Defense  Authorization  Acts  of  Fiscal  Years  2006,  2007,  and  2008  changed 
certification  standards,  requiring  additional  certification  requirements  for  major  defense 
acquisition  programs  before  proceeding  to  milestones  A  or  B.31  In  addition  USD  (AT&L) 
issued  a  memo  in  May,  2007  implementing  Program  Manager  Agreements  to  establish  an 
annual  contract  between  the  program  manager  and  the  acquisition  and 
requirements/resource  officials.3’  These  are  binding  agreements  that  detail  the  progress  the 
program  is  expected  to  make  during  the  year  and  the  resources  the  program  will  be  provided 
to  reach  these  goals.  USD  (AT&L)  also  requires  PMs  to  sign  tenure  agreements  with  the  SAE 
stipulating  that  their  MDAP  tenure  will  correspond  to  the  next  major  milestone  review 
closest  to  4  years.  In  our  opinion,  when  certifications  are  used  in  conjunction  with  Program 
Manager  Agreements,  they  provide  more  program  visibility  and  a  stronger  means  for  holding 
the  PM  and  the  other  key  management  officials  accountable  for  the  decisions  they  make 
managing  their  programs.  Furthermore,  DOD’s  strategy  to  improve  acquisition  outcomes 
rests  not  only  with  providing  support  for  individual  PMs,  but  more  importantly  on  creating  a 
department-wide  environment  that  promotes  stability,  accountability,  and  more  informed 
decision  making.33  Many  of  the  experts  we  interviewed  cited  the  new  legislation — which 
requires  milestone  A  and  B  certifications — as  adequate  to  meet  the  needs  that  this  change 
targets.  We  agree. 

Modifying  the  Role  Played  by  Chiefs  of  Staff  of  the  Armed  Forces  in  the  Requirements, 

Resource  Allocation,  and  Acquisition  Processes 

It  is  difficult  to  say  whether  the  potential  change,  as  presented,  will  generate  better 
acquisition  outcomes.  In  our  view,  this  potential  change  requires  further  definition  and  more 


31 10  USC  §  2366A  and  §  2366B  have  certification  criteria  for  both  milestone  A  and  B  reviews. 

32USD  (AT&L)  Memorandum,  Program  Management  Tenure  and  Accountability,  May  25,  2007. 

33GAO,  Defense  Acquisitions:  Department  of  Defense  Actions  on  Program  Manager  Empowerment 
and  Accountability,  GAO-08-62R  (Washington,  D.C.:  March  11,  2008). 
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study.  It  is  unclear  which  roles  of  the  Chiefs  of  Staff  of  the  Armed  Forces  should  be  changed 
and  how  these  roles  should  be  modified  to  improve  acquisition  outcomes.  However,  we  note 
that  the  Beyond  Goldwater-Nichols  body  of  work  concluded  that  a  consequence  of 
implementing  the  1986  Goldwater-Nichols  legislation  is  that  the  military  service  Chiefs  of 
Staff  no  longer  play  a  large  role  in  managing  the  execution  of  their  acquisition  programs.34 
According  to  the  DAPA  panel,  the  DSB  2005  Summer  Study  on  Transformation,  and  the 
Beyond  Goldwater-Nichols  body  of  work,  restoring  the  authority  to  manage  the  execution  of 
their  acquisition  programs  to  the  military  service  chiefs  could  in  turn  restore  accountability 
for  acquisition  results.  A  chief  argument  of  each  of  these  studies  is  that  by  realigning 
acquisition  programs  through  a  uniformed  services  chain  of  command,  the  services’ 
acquisition  organizations  would  be  more  streamlined  and  accountability  would  be  easier  to 
enforce.  A  large  majority  of  the  experts  we  interviewed  who  commented  on  this  potential 
change  (35  of  43)  told  us  that  modifying  the  role  of  the  Chiefs  of  Staff  of  the  Armed  Forces  to 
provide  them  authority  to  manage  the  execution  of  their  acquisition  programs  would  not 
improve  the  acquisition  process.  Many  of  the  experts  told  us  the  military  services  already 
have  control  over  the  requirements  and  resource  allocation  processes,  and  should  not  also 
have  control  over  the  acquisition  process.  On  the  other  hand,  some  experts  commented  that 
changing  the  roles  of  the  military  Chiefs  of  Staff  might  improve  the  process  if  done  in 
conjunction  with  making  the  combatant  commanders  responsible  for  determining 
requirements,  a  condition  that  would  take  place  if  the  potential  change  to  allow  combatant 
commanders  more  influence  concerning  their  long-term  needs  is  made. 

Changes  Needed  in  Acquisition  Environment,  Culture,  and  Incentives 

Our  discussions  with  acquisition  experts  indicated  that  the  potential  changes  may  not 
achieve  the  desired  improvement  in  acquisition  outcomes  unless  they  are  accompanied  with 
changes  in  the  overall  acquisition  environment,  its  culture,  and  the  incentives  provided  for 
success.  Specifically,  some  of  the  DOD  executives  we  interviewed — both  current  and 
former — emphatically  stated  that  these  changes  will  not  improve  the  acquisition  system 
unless  the  department  can  significantly  improve  its  prioritization  of  needs  so  that  there  are 
fewer  programs  competing  for  available  dollars.  Other  experts  we  interviewed  questioned  the 
potential  of  all  seven  changes  to  improve  acquisition  outcomes.  A  few  of  the  experts  believe 
some  process  stakeholders  involved  in  setting  requirements,  providing  funding,  or  executing 
programs  would  not  agree  to  the  changes  because  they  like  the  process  as  it  is  or  do  not  want 
their  roles  in  the  process  to  change.  One  DOD  acquisition  official  told  us  that  DOD’s 
acquisition  policies  are  good;  they  just  are  not  implemented  as  intended.  Another  expert — a 
former  high-ranking  DOD  executive — categorically  stated  that  the  acquisition  system  is  a 
system  in  equilibrium  with  rules  and  regulations  that  allow  leadership  to  deviate  from  the 
process  when  they  want.  Until  members  of  Congress  and  DOD  leadership  collectively  realize 
this  is  the  case  as  well  as  the  fact  that  they  all  like  the  system  as  it  is,  nothing  will  change. 
Many  believe  it  will  take  strong,  consistent  strategic  leadership  to  change  the  existing  culture 
and  environment. 

Conclusions 


We  believe  that  six  of  the  seven  potential  changes  to  DOD’s  acquisition  framework  could 
improve  acquisitions  outcomes.  Although  many  of  the  experts  we  interviewed  believed  the 
potential  changes  may  not  achieve  the  desired  improvement  in  acquisition  outcomes,  they 
provided  a  wide  range  of  opinions  concerning  the  pros  and  cons  of  each  potential  change. 
Nonetheless,  based  on  the  results  of  this  review  and  our  work  and  observations  over  the 
years  of  the  DOD  acquisition  systems’  imperviousness  to  change,  we  do  not  believe  that  the 
changes  are  likely  to  achieve  the  desired  results  unless  they  are  (1)  implemented  together 


"Goklwater  Nichols  Department  of  Defense  Reorganization  Act  of  1986,  Pub.  L.  No.  99-433. 
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along  with  other  more  systemic  and  sweeping  changes  and  (2)  accepted  by  all  of  the 
stakeholders  and  process  owners  in  this  complex  framework  of  requirement  setting,  funding, 
and  program  execution.  We  point  specifically  to  comments  made  by  those  current  and  former 
DOD  executives  who  know  the  environment  best.  Some  of  these  experts  stated  that  (1)  the 
requirements  setting  process  (JCIDS)  takes  too  long;  generally  produces  requirements  for 
capabilities  that  aren’t  achievable  in  the  short  term,  and  often  doesn’t  provide  optimal  results; 
(2)  the  funding  process  historically  accepts  cost  and  schedule  estimates  based  on  optimistic 
assumptions  and  little  knowledge  about  the  resources  needed  to  develop  the  weapon  system 
required;  and  (3)  the  acquisition  process  moves  forward  with  those  poor  estimates,  often 
adding  requirements  and  increasing  cost  as  the  development  program  progresses.  Finally, 
some  of  the  experts  we  interviewed  pointed  to  an  overall  lack  of  discipline  in  the  process 
when  it  comes  to  making  resource-constrained  decisions.  All  of  these  factors  sustain  an 
environment  of  high  risk,  low  reward,  and  poor  outcomes  in  the  form  of  higher  costs,  late 
deliveries,  and  less  than  optimal  capabilities  fielded  to  the  warfighter. 

Potential  incremental  changes  to  the  process  are  not  likely  to  produce  lasting  and  meaningful 
improvement  until  the  acquisition  environment  and  all  of  its  process  owners  accept  that  risk 
in  product  development  must  be  lowered  by  prioritizing  the  needs  of  national  security  jointly 
and  with  discipline;  systems  engineering  knowledge  is  used  to  constrain  requirements  prior 
to  program  initiation;  funding  programs  are  based  on  knowledge  and  independent  cost 
estimates  with  high  levels  of  confidence;  and  evolutionary,  knowledge-based  practices  and 
processes  to  reduce  technology,  design,  and  manufacturing  risk  are  employed.  Nonetheless, 
we  believe  the  potential  changes  discussed  in  this  report  could  have  significant  positive 
effect,  but  not  if  the  environment,  its  incentives,  and  the  culture  continue  to  resist  change. 

Agency  Comments  and  Our  Evaluation 

We  provided  a  draft  of  this  correspondence  to  DOD  for  comment.  The  department  did  not 
provide  a  written  response,  but  provided  technical  comments  which  we  have  incorporated  as 
appropriate. 


We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense;  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics;  the  Chairman  of  the  Joint  Chiefs  of  Staff; 
and  the  Secretaries  of  the  Air  Force,  Army,  and  Navy.  This  report  will  also  be  available  at  no 
charge  on  GAO’s  Web  site  at  http://www.  gao.  gov.  If  you  have  any  questions  about  this  report 
or  need  additional  information,  please  contact  me  at  (202)  512-4841  or  sullivanm@gao.gov. 
Contact  points  for  our  Offices  of  Congressional  Relations  and  Public  Affairs  may  be  found  on 
the  last  page  of  this  report. 

Key  contributors  to  this  report  were  Bruce  Thomas,  Assistant  Director;  Claudia  Dickey, 
Analyst-in-Charge;  Greg  Campbell;  Tina  Cheng;  Rosa  Johnson;  John  Krump;  Jean  McSween; 
Robert  Miller,  and  J.  Andrew  Walker. 


Michael  J.  Sullivan,  Director 
Acquisition  and  Sourcing  Management 
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Enclosure  I:  Scope  and  Methodology 


The  2008  National  Defense  Authorization  Act  (NDAA)  directed  GAO  to  report  on  the 
feasibility  and  advisability  of  seven  potential  changes  to  improve  the  Department  of 
Defense’s  (DOD)  acquisition  framework  for  Major  Defense  Acquisition  Programs  (MDAPs).35 
See  enclosure  II  for  a  list  of  the  seven  potential  changes.  Our  objective  was  to  determine  the 
feasibility  and  advisability  of  the  potential  modifications. 

To  determine  the  feasibility  and  advisability  of  the  seven  potential  changes  included  in  the 
mandate,  we  considered  the  results  of  GAO’s  own  body  of  work  and  relevant  works 
produced  by  DOD  and  others,  and  the  results  of  our  interviews  with  53  acquisition  experts. 
We  identified  and  analyzed  recommendations  from  prior  GAO  and  DOD  work  that  address 
Congress’s  potential  changes,  and  considered  the  opinions  of  the  53  experts  we  interviewed. 
Enclosure  IV  provides  a  list  of  DOD  acquisition  studies  and  GAO  reports  used  to  inform  our 
work.  We  reviewed  relevant  law  and  DOD  policies,  directives,  and  guidance  as  well  as  the 
history  of  the  department’s  DAS  and  management  framework. 

We  conducted  semistructured  interviews  with  53  identified  experts.  We  selected  the  experts 
from  among  currently  serving  senior  DOD  acquisition  officials,  senior  military  officers 
involved  in  setting  requirements  for  the  joint  staff,  the  military  services,  the  Combatant 
Commands  (COCOM),  individuals  who  formerly  served  as  senior  DOD  acquisition  officials, 
participants  in  previous  reviews  of  the  organization  and  structure  of  DOD  for  the  acquisition 
of  major  weapon  systems,  academicians,  and  other  subject  matter  experts.  The  interview 
questions  were  designed  to  determine  the  expert’s  opinions  concerning  the  seven  potential 
changes,  as  well  as  determine  other  systemic  problems  in  DOD’s  overall  acquisition 
management  framework  and  other  potential  changes  that  might  result  in  better  acquisition 
outcomes.  We  pretested  the  questions  during  initial  interviews  with  DOD  officials  located  in 
Dayton,  Ohio  and  at  the  Pentagon.  Based  on  the  results  of  these  interviews,  we  refined  our 
questions  to  ensure  they  were  open  ended  and  allowed  the  respondents  the  opportunity  to 
provide  not  only  opinions  on  the  potential  changes,  but  also  their  reasoning.  Additionally,  the 
nature  of  the  questions  allowed  the  experts  to  provide  additional  opinions  concerning 
problems  with  DOD’s  acquisition  processes  and  potential  solutions.  To  summarize  and 
analyze  the  information  obtained  during  our  interviews,  we  recorded  and  verified  the  experts’ 
responses  in  a  Web-based  data  collection  instrument.  The  content  for  the  responses  were 
then  coded  and  verified  to  determine  the  extent  to  which  the  experts  believed  the  seven 
potential  changes,  would,  would  not,  or  might  improve  DOD’s  acquisition  processes,  and  to 
identify  other  potential  challenges  and  solutions. 

We  conducted  this  performance  audit  from  January  2008  to  February  2009  in  accordance 
with  generally  accepted  government  auditing  standards.  Those  standards  require  that  we  plan 
and  perform  the  audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis 
for  our  findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit 
objectives. 


3BPub.  L.  No.  110-181. 
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Enclosure  II:  Potential  Modifications  to  the  DOD  Acquisition  Process  from  the 
National  Defense  Authorization  Act  for  2008 


1.  Revising  the  acquisition  process  for  major  defense  acquisition  programs  by 
establishing  shorter,  more  frequent  acquisition  program  milestones; 

2.  Requiring  certifications  of  program  status  to  the  defense  acquisition  executive  and 
Congress  prior  to  milestone  approval  for  major  defense  acquisition  programs; 

3.  Establishing  a  new  office  (to  be  known  as  the  “Office  of  Independent  Assessment”) 
to  provide  independent  cost  estimates  and  performance  estimates  for  major  defense 
acquisition  programs; 

4.  Requiring  the  milestone  decision  authority  for  a  major  defense  acquisition  program  to 
specify,  at  the  time  of  Milestone  B  approval,  or  Key  Decision  Point  B  approval,  as 
applicable,  the  period  of  time  that  will  be  required  to  deliver  an  IOC  to  the  relevant 
combatant  commanders; 

5.  Establishing  a  material  solutions  process  for  addressing  identified  gaps  in  critical 
warfighting  capabilities,  under  which  process  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  circulates  among  the  military  departments  and 
appropriate  Defense  Agencies  a  request  for  proposals  for  technologies  and  systems  to 
address  such  gaps; 

6.  Modifying  the  role  played  by  chiefs  of  staff  of  the  Armed  Forces  in  the  requirements, 
resource  allocation,  and  acquisition  processes;  and 

7.  Establishing  a  process  in  which  the  commanders  of  the  combatant  commands  assess, 
and  provide  input  on,  the  capabilities  needed  to  successfully  accomplish  the  missions 
in  the  operational  and  contingency  plans  of  their  commands  over  a  long-term 
planning  horizon  of  15  years  or  more,  taking  into  account  expected  changes  in 
threats,  the  geo-political  environment,  and  doctrine,  training  and  operational 
concepts. 
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Enclosure  III:  Comparison  of  DOD’s  Current  and  Prior  Acquisition  Processes 


Current  DOD  Process  (as  of  Dec  2008) 
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Source:  GAO  analysis  of  DOD's  current  and  prior  acquisition  process. 


Major  differences  between  the  2003  and  2008  versions  of  the  DOD  instruction  5000.02  are: 

•  The  Materiel  Development  Decision  (MDD)  replaces  the  Concept  Decision.  A  MDD  is 
required  regardless  of  where  the  program  intends  to  enter  the  acquisition  process. 

•  The  Materiel  Solution  Analysis  (MSA)  Phase  replaces  the  Concept  Refinement  Phase. 
The  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  process  no 
longer  includes  an  Analysis  of  Materiel  and  Non-Materiel  Alternatives.  Non-materiel 
solutions  will  be  handled  via  JCIDS;  however,  all  analysis  of  alternative  materiel 
solutions  will  be  accomplished  by  the  Analysis  of  Alternatives  (AO A)  during  MSA. 
The  Milestone  Decision  Authority  (MDA)  will  approve  the  materiel  solution  at 
milestone  A. 

•  The  Technology  Development  Phase  (TD)  now  includes  a  mandatory  requirement  for 
competitive  prototyping  of  the  system  or  key-system  elements.  A  Preliminary  Design 
Review  (PDR)  may  be  conducted  for  the  candidate  designs,  and  a  PDR  report  will  be 
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provided  to  the  MDA  with  recommended  requirements  trades.  The  final  Capabilities 
Development  Document  should  contain  trade-offs  determined  during  the  TD  phase. 

•  The  Engineering  &  Manufacturing  Development  (EMD)  phase  replaces  System 
Development  and  Demonstration.  The  two  major  efforts  have  been  renamed.  The 
Program  Manager  must  provide  a  PDR  report,  and  must  provide  a  Critical  Design 
Review  (CDR)  report  to  the  MDA. 

•  A  Post-CDR  Assessment  replaces  the  Design  Readiness  Review.  The  MDA  will 
determine  if  the  results  of  the  CDR  warrant  continuing  EMD  to  milestone  C. 
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Studies  on  POD  Acquisition 

President’s  Blue  Ribbon  Commission  on  Defense  Management,  A  Formula  for  Action:  A 
Report  to  the  President  on  Defense  Acquisition.  The  Packard  Commission.  Washington, 

D.C.:  April  1,  1986.  http ://www.ndu. edu/librarv/pbrc/pbrc.html 

Defense  Acquisition  Performance  Assessment  Report.  Department  of  Defense  Acquisition 
Performance  Assessment  Panel.  Washington,  D.C.:  January  27,  2006. 
www.aca.osd.mil/dapanroiect 

Beyond  Goldwater-Nichols:  Defense  Reform  for  a  New  Stra  tegic  Era,  Phase  1  Report. 

Center  for  Strategic  and  International  Studies.  Washington,  D.C.:  March  2004. 
www.  csis.  org/isp/b  gn 

Beyond  Goldwater-Nichols:  U.S.  Government  and  Defense  Reform  for  a  New  Strategic  Era, 
Phase  2  Report.  Center  for  Strategic  and  International  Studies.  Washington,  D.C.:  July  2005. 
www.  csis.  org/isp/b  gn 

The  Future  of  the  National  Guard  and  Reserves:  Beyond  Goldwater-Nichols:  Department  of 
Defense  Acquisition  and  Planning,  Programming,  Budgeting,  and  Execution  System 
Reform,  Phase  3  Report.  Center  for  Strategic  and  International  Studies.  Washington,  D.C.: 
August  2006.  www.csis.org/isp/bgn 

Beyond  Goldwater-Nichols:  Department  of  Defense  Acquisition  and  Planning, 
Programming,  Budgeting,  and  Execution  System  Reform,  An  Annotated  Brief.  Center  for 
Strategic  and  International  Studies.  Washington,  D.C.:  August  1,  2006.  www. csis. org/isp/bgn 

Transformation:  A  Progress  Assessment  Volume  I.  Defense  Science  Board,  2005  Summer 
Study.  Washington,  D.C.:  February  2006.  http://www.acq.osd.mi1/dsb/rcpoi1s.hlm#2005 

Defense  Acquisition  Transformation,  Report  to  Congress.  John  Warner  National  Defense 
Authorization  Act  Fiscal  Year  2007,  Section  807.  Washington,  D.C.:  March  2008 
www.acq.osd.mil/al/docs/804  defense  acquisition  transformation  report  mar  2008.pdf 

Report  to  Congress  on  Program  Manager  Empowerment  and  Accountability .  Department  of 
Defense.  Washington,  D.C.:  August  2007. 

www.acq.osd.mil/at/docs/Program  Manager  Empowerment  and  Accountabilitv.pdf 

Joint  Defense  Capabilities  Study,  Improving  DOD  Strategic  Planning,  Resourcing  and 
Execution  to  Satisfy  Joint  Capabilities.  Department  of  Defense.  Washington,  D.C.:  January 
2004. 

www.acq.osd.mil/actd/articlcs/JointDcfcnscCnpabiliticsStiidvFinalReDort  Januarv2004.pdf 
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